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Abstract 



Problem 



To easily and accurately correct for red-eye in flash 
photography. 



The designation of areas surrounding a pair of eyes and the 
specification of a point between the eyes are carried out (SI), 
the characteristic quantity of the color at the specified point 
is calculated (S2) , a skin color area is extracted from the 
designated area (S3), an average value of the chromaticity of the 
red color within the skin color area is calculated (S4), an area 
circumscribing the skin color area is designated (S5) , this 
designated area 'is further divided into small areas (S6) , pupil 



Solution 
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candidate areas are selected from the small areas (S7), red-eye 
pixels in the pupil candidate areas are extracted (S8) in order 
to correct said red-eye pixels (S9) . 
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Key: SI 


Designation of an area surrounding a pair of eyes. 




A point between the eyes is specified. 


S2 


Characteristic quantity of the color at the specified 




point is calculated. 


S3 


Extraction of skin color areas. 


S4 


Calculation of the average value (Rskin) of the 




chromaticity of the red color in the skin color area. 


S5 . 


An area circumscribing the skin color area is 




designated. 


S6 


Division into small areas. 
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57 Pupil candidate areas are selected from the small 
areas . 

58 Red-eye pixels within the pupil candidate areas are 
extracted. 

59 Red-eye pixels are corrected. 

Claims 



1. Image processing method characterized in that an area 
which may contain .1 or more poor color tones in the pupils is 
designated in a photo image which contains pupils in the subject, 

the aforementioned designated area is divided into small 
areas, 

small areas with a high possibility of containing pupils 
with poor color tones are selected as pupil candidate areas based 
on color data and positional data of the aforementioned small 
areas, 

pixels with poor color tones are extracted from the 
aforementioned selected pupil candidate areas, and 

the aforementioned extracted pixels are corrected for poor 
color tones. 

2. Image processing method described in Claim 1 and 
characterized in that an area surrounding a pair of pupils is 
designated for the aforementioned area possibly containing pupils 
with poor color tones, the center point of said designated area 
is calculated, and the division into small areas and the 
selection of pupil candidate areas are carried out based on color 
data and positional data using said center point as a reference. 

3. Image processing method described in Claim 1 and 
characterized in that the aforementioned designation of the area 
surrounding the pair of pupils is done by specifying a point near 
the center between the pupils and by designating an area of a 



prescribed area around said specified point, and the division 
into small areas and the selection of the pupil candidate areas 
are carried' out based on color data and positional data using the 
aforementioned specified point as a reference. 

4. Image processing method described in Claim 1 and 
characterized in that to designate the aforementioned area where 
poor color tones may be present in the pupils, an area 
surrounding a pair of pupils where 1 or more poor color tones may 
be present is designated, a point near the center between the 
pupils is specified, and the division into small areas and the 
selection of pupil candidate areas are carried out based on color 
data and positional data using said center point as a reference. 

5. Image processing method described in one of Claims 

1 through 4 and characterized in that the aforementioned division 
into small areas is done by extracting edge pixels from the 
aforementioned designated area based on color data and by 
dividing the designated area into areas surrounded by said edge 
pixels . 

6. Image processing method described in one of Claims 
1 through 5 and characterized in that the aforementioned 
selection of pupil candidate areas is done by using evaluation 
functions indicating pupil probability based on color data and 
positional data after 2 of the aforementioned small areas are 
combined into 1, respectively, and by selecting the combination 
which shows the highest value. 

7. Image processing method described in one of Claims 
1 through 6 and characterized in that the aforementioned 
correction to the pixels with poor color tones is carried out 
based on the brightness data for the color elements other than 
red contained in the pixels with poor color tones. 



8. Image processing method described in one of Claims 
1 through 7 and characterized in that the aforementioned 
correction of pixels with poor color tones is carried out based 
on the data for the colors selected by the operator from multiple 
color samples. 

9. Image processing method described in one of Claims 
1 through 8 and characterized in that to carry out the 
aforementioned correction of pixels with poor color tones, the 
range of the corrections of pixels with poor color tones is made 
adjustable for the operator through the arbitrary setting of the 
threshold with regard to color data for the extraction of pixels 
with poor color tones. 

Detailed explanation of the invention 
[0001] 

Technical field of the invention 

The present invention pertains to an image processing method 
for the correction of poor color tones in pupils in a subject 
image due to the emission of a flash in color photography. 

[0002] 
Prior art 

Several techniques for correcting the aforementioned poor 
color tones, so-called red-eye, have been suggested (Refer to 
U.S. Patent No. 5130789, Japanese Kokai Patent Application No. 



Hei 7(19951-72537, etc.). In the technique disclosed in U.S. 
Patent No. 51307 89, a target area around the eyes is surrounded 
by a rectangular mask, several points in red-eye areas are 
specified using a pointer, the red-eye areas are designated based 
on color data, and corrections are made once the target pixels 
are determined to possibly cause red-eye. At that time, the 
correction amount is altered according to the distance of the 
area from the edge . 

[0003] 

Also, in the technique disclosed in Japanese Kokai Patent 
Application No. Hei 7 ( 1995] -72537, a target eye area is 
designated, and after candidate pixels are selected based on 
color data and labeled, whether they cause red-eye or not is 
determined by considering data from respective labels. In 
addition, in terms of the correction method, corrections are made 
at different rates to the main part of the red-eye and to the 
surrounding part. 

[0004] 

Problems to be solved by the invention 

With respect to the technique in U.S. Patent No. 5130789, a 
burden is imposed on the operator in that several points must be 
specified in the -red-eye areas in addition to surrounding the 
target areas around the eyes with the rectangular mask. With 
respect to the technique in Japanese Kokai Patent Application 
No. Hei 7 [1995] -72537, although the candidate areas are 
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designated through the comparison of multiple color data with the 
threshold, because the color data, particularly for the areas 
surrounding' the red-eye, differ significantly according to the 
differences in terms of the photographic scene and the 
individuals, it is difficult to use the same conditions for 
setting, causing lower efficiency. Also, the correction method is 
complicated in that main parts must be separated from surrounding 
areas to which corrections are made at different rates. 

[0005] 

The present invention was made considering these 
conventional problems, and its purpose is to present an image 
processing method by which red-eye correction can be done easily 
and accurately. 

[0006] 

Means to solve the problems 

To this end, as shown in the block diagram in Figure 1, the 
invention in Claim 1 is characterized in that an area which may 
contain 1 or more poor color tones in the pupils is designated in 
a photo image which contains pupils in the subject, the 
aforementioned designated area is divided into small areas, small 
areas with a high possibility of containing pupils with poor 
color tones are selected as pupil candidate areas based on color 
data and positional data of the aforementioned small areas, 
pixels with poor color tones are extracted from the 
aforementioned selected pupil candidate areas, and the 
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aforementioned extracted pixels are corrected for poor color 
tones . 

[0007] 

Also, the invention in Claim 2 is characterized in that an 
area surrounding a pair of pupils is designated for the 
aforementioned area possibly containing pupils with poor color 
tones, the center point of said designated area is calculated, 
and the division into small areas and the selection of pupil 
candidate areas are carried out based on color data and 
positional data using said center point as a reference. 

[0008] 

Also, the invention in Claim 3 is characterized in that the 
aforementioned designation of the area surrounding the pair of 
pupils is done by specifying a point near the center between the 
pupils and by designating a prescribed area around said specified 
point, and the division into small areas and the selection of the 
pupil candidate areas are carried out based on color data and 
positional data using the aforementioned specified point as a 
reference. 

[0009] 

Also, the invention in Claim 4 is characterized in that to 
designate the aforementioned area where poor color tones may be 
present in the pupils, an area surrounding a pair of pupils where 
1 or more poor color tones may be present is designated, a point 



10 



near the center between the pupils is specified, and the division 
into small areas and the selection of pupil candidate areas are 
carried out based on color data and positional data using said 
center point as a reference. 

[0010] 

Also, the invention in Claim 5 is characterized in that the 
aforementioned division into small areas is done by extracting 
edge pixels from the aforementioned designated area based on 
color data and by dividing the designated area into areas 
surrounded by said edge pixels. Also, the invention in Claim 6 is 
characterized in that the aforementioned selection of pupil 
candidate areas is done by using evaluation functions indicating 
pupil probability based on color data and positional data after 
2 of the aforementioned small areas are combined into 1, 
respectively, and by selecting the combination which shows the 
highest value of all. 

[0011] 

Also, the invention in Claim 7 is characterized in that the 
aforementioned correction of the pixels with poor color tones is 
carried out based on the brightness data for the . color elements 
other than red contained in the pixels with poor color tones. 
Also, the invention in Claim 8 is characterized in that the 
aforementioned correction of the pixels with poor color tones is . 
carried out based on the color data selected by the operator from 
multiple color samples. 



[0012] 



Also, the invention in Claim 9 is characterized in that to 
carry out the aforementioned correction of the pixels with poor 
color tones, the range of corrections for the pixels with poor 
color tones is made adjustable for the operator through arbitrary 
setting of a threshold with regard to the color data for the 
extraction of pixels with poor color tones. 

[0013] 

Effect of the invention 

With the invention pertaining to Claim 1, because the pupil 
candidate areas can be selected highly accurately from the 
respective small areas obtained by dividing the area possibly 
containing pupils with 1 or more poor color tones, red-eye can be 
corrected accurately and pixels with poor color tones in the 
pupil candidate areas can be corrected. 

[0014] 

Also, with the invention pertaining to Claims 2 through 4, 
the pupil candidate areas can be selected highly accurately with 
a minimum burden on the operator. Also, with the invention 
pertaining to Claim 5, because the division into small areas with 
almost the same color tone base can be achieved by dividing the 
target area into areas surrounded by the edge pixels, extraction 
of the pixels with poor color tones from the small areas becomes 
easier. 
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[0015] 

Also, with the invention pertaining to Claim 6, because 
evaluations are made on combined sets respectively comprising 2 
small areas, pupil candidate areas can be selected highly 
accurately by means of evaluation functions utilizing symmetry, 
especially when red-eye is present in both pupils. Also, with the 
invention pertaining to Claim 7, based on the brightness data for 
color elements other than red, for example, blue and green, 
brightness of red or of red, blue, and green can be matched with 
the quality of said brightness in order to correct for a natural 
look. 

[0016] 

Also, with the invention pertaining to Claim 8, the operator 
can make corrections in order to obtain desired colors by making 
selections from color samples. Also, with the invention 
pertaining to Claim 9, because the threshold is provided, the 
operator can expand or reduce the areas to be corrected according 
to the degree of poor color tone as determined by the operators- 
variations in the color data, for the areas with poor color tones 
and the surrounding areas due to the differences in terms of the 
photographic scene and the individuals, can be absorbed. 



[0017] 



Embodiments' of the invention 

Embodiments of the present invention will be explained below 
with reference to figures. Figure 2 shows the system 
configuration of an embodiment. Digital image data for a color 
image captured through a still video camera or digital image data 
read using a scanner from a color image captured through a silver 
halide camera and printed paper and developed on a film are 
stored in a memory unit (1) such as an optical disk. 

[0018] 

The image data stored in the aforementioned memory unit (1) 
are read by a control unit (2), and an image is displayed on a 
monitor (3) . While looJcing at the image displayed on the 
aforementioned monitor, the operator designates image areas where 
red-eye due to flash photography is present in the subject image 
in the following manner in order for the aforementioned control 
unit (2) to extract the pixels causing red-eye from said areas, 
make corrections, and display the corrected image on the 
monitor (3) . 

[0019] 

The red-eye correction routine pertaining to the present 
invention will be explained below with reference to the flow 
charts in Figure 3 and following. Figure 3 shows the red-eye 
correction routine. In Step 1 (indicated by S in the figure and 



following), a rectangular area surrounding a pair of eyes is 
designated in the subject image containing the pupils and 
displayed oh the monitor (2), and a point near the center between 
the pair of eyes is specified as a reference point for color data 
and positional data to be described later (refer to Figure 6) . 

[0020] 

As the method for setting the aforementioned target area and 
the center point, both the target area and the center point may 
be specified by the operator; or only the target area is 
designated and the center point is then calculated (for example, 
the center of the designated rectangular area is calculated as 
the center point) ; or, only the center point is specified, and a 
prescribed area around said point may be taken as the target 
area. For typical red-eye correction, the first method is most 
suitable for manually carrying out the designation of the target 
area and the specification of the center point. 

[0021] 

The characteristic of the present invention is that an area 
surrounding the pair of pupils is designated, for example, even 
when only 1 pupil of the pair of pupils contains poor color 
tones. In addition, when both pupils have the red-eye effect, 
2 red-eye effects can be corrected simultaneously with only one 
designation of the area. For example, in the technique disclosed 
in aforementioned U.S. Patent No. 5130789, because the area 
designation is carried out for each pupil, the operator's 
involvement is increased. 
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[0022] 

In Step 2, color data, for example, the total value 
(R + G + B) of the respective values of the brightness of R 
(red), G (green), and B (blue), at the point specified in 
aforementioned Step 1 are calculated as the characteristic 
quantity. In Step 3, those pixels with a characteristic quantity 
different from the characteristic quantity at the aforementioned 
specified point only wi t-hin a prescribed threshold are determined 
as skin color pixels, and an area containing said skin color 
pixels is extracted. That is, because the aforementioned 
specified point, the point near the center between the eyes, is 
located at the base of the nose, the color at said point is 
assumed to be skin color, so that an area with a color close to 
the color at said point can be assumed to contain pixels with the 
skin color. In addition, hue and chroma, which are general color 
data, may be used as characteristic quantities. In this way, the 
pixels which fall under the aforementioned threshold are labeled, 
and the labeled area containing the specified point is extracted 
as a skin color area. 

[0023] 

In Step 4, an average value (Rskin) of the chromaticity 
(= R/ (R + G + B) ) of the red color in the aforementioned skin 
color area is obtained. In Step 5, in order to reduce the amount 
of the following calculation and the amount of the data necessary 
for said calculation, a rectangular area (an area with long sides 
containing the point between the eyes and circumscribing both 



eyes simultaneously) circumscribing the skin color area extracted 
in aforementioned Step 3 is designated as a new target area. The 
designation' of said rectangular area is done automatically. 
Although it may be done manually, if it is done automatically, a 
situation in which the operator designates a large area 
containing areas other than the face can also be dealt with. 

[0024] 

In Step 6f the aforementioned target area is divided into 
small areas. The subroutine for said division into small areas 
will be explained according to the flow chart in Figure 4. In 
Step 11, edge pixels are obtained which are within the 
aforementioned designated area. As for the method for obtaining 
the edge pixels, the sum of the absolute values of the 
differences in brightness between the target pixels and the 
respective surrounding 4 pixels is obtained using the brightness 
(R + G + B) , for example, as the characteristic quantity; and if 
said sum is equal to or greater than the prescribed threshold, 
the target pixels are determined to be edge pixels. In addition, 
an edge detection filter, such as a Sobel operator, Prewitt 
operator, and etc., may be utilized. 

[0025] 

In Step 12, the edge pixels detected in aforementioned Step 
11 are labeled. In Step 13, the pixel surrounded on at least 
3 sides by edge pixels is changed to the same label as that of 
the aforementioned edge pixels. In other words, the area 
surrounded by the edge pixels adopt the same label. In this way. 



the designated area is divided into small areas with the same 
labels. 

[0026] 

Returning to the main routine in Figure 3, in Step 7, an 
area with a high probability of containing pupils with poor color 
tones is selected as a pupil candidate areas from the 
aforementioned divided small areas. The subroutine for the 
aforementioned selection of pupil candidate areas in Step 7 will 
be explained with reference to the flow chart in Figure 5. 

[0027] 

In the present embodiment, 2 of the aforementioned divided 
small areas are combined into 1, respectively, and pupil 
probability is obtained through the calculation of the following 
characteristic quantity for all combinations. In Step 21, 
chromaticity (= R/ (R + G + B) ) of the red color in respective 
small areas is obtained, and the maximum value (Ri) of the 
chromaticity of the red color over the areas and the position 
(Xi, Yi) of the pixel with the aforementioned maximum value (Ri) 
on the X and Y axes when the point near the center between the 
eyes specified in aforementioned Step 1 is used as the origin are 
used to indicate the characteristic quantity. Here, i represents 
a value for the identification of the small area. 
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[0028] 

For example, when combining small areas (1) and (2), a pupil 
candidate area is selected in the following manner. In Step 22, 
characteristic quantity of color data (C12) and characteristic 
quantity of positional data (P12) are obtained using the 
following equations . 

Characteristic quantity of color data (C12) = Rl + R2 

Characteristic quantity of positional data 

(P12) = XI + X2 + Yl + Y2 

Here, a greater characteristic quantity of color data and a 
smaller characteristic quantity of positional data means a higher 
pupil probability. Thus, in Step 23, the characteristic quantity 
(E12) indicating the pupil probability is set as E12 = C12 - P12, 
for example, and the combination of small areas highest in said 
value is extracted as a pupil candidate area with a high 
possibility of containing at least one pupil area with poor color 
tones . 

[0029] 

Here, the reason why the smaller the characteristic quantity 
of the positional data is, the higher the pupil probability 
becomes is that when red-eye is present in both eyes, the value 
becomes smaller because the calculation is based on XI, either 
the left or the right side across the point between the eyes is 
treated as + and X2 on the other side is treated as -. Also, the 
reason why the case in which only one pupil contains red-eye can 
be handled is that when the combination of both pupils is 
selected, the values of the positional data (P12) become smaller 
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even when one pupil does not contain red-eye. Also, even when 
only one pupil has red-eye, the characteristic quantity of the 
color data is greater in comparison to a combination without 
red-eye . 

[0030] 

Returning to the main routine in Figure 3, in Step 8, pixels 
with poor color tones (red-eye) are extracted from the pupil 
candidate areas extracted in Step 7. The pupil extraction method 
is as follows. If the chromaticity (= R/ (R + G + B) ) of the red 
color of the target pixels is greater than the average value 
(Rskin) of the skin color area obtained in Step 2, and it is also 
greater than the value obtained by subtracting the preset 
threshold (TH) from the characteristic quantity (Ri) of the color 
data on the target area, they are determined to be red-eye 
pixels. In other words, the pixels which meet the following 
equation are considered to be red-eye pixels. 

[0031] 

R/{R + G + B) > Rskin n R/ (R + G + B) > Ri - TH 
In Step 9, the red-eye pixels extracted in Step 8 are 
corrected to achieve appropriate colors. In this case, because a 
very unnatural look will result if the pixels are all corrected 
to have the same pixel value, brightness data other than for the 
color red is utilized. For example, when the digital values of R, 
G, and B are R = 200, G = 50, and B = 30, respectively, 
correction is made with a gray color which matches the lowest 
value of the digital values. In this case, because the correction 
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can be made to achieve a pupil image which reflects the 
brightness of R = G = B = 30 to some extent, a natural look will 
be obtained. 

[0032] 

Otherwise, in order to handle the case in which the original 
color of the pupils is not gray, the values of G and B are kept 
constant when the digital values of R, G, and B are R = 200, 
G = 50, and B = 30, respectively, as in the aforementioned case, 
and R is matched with the larger value of G and B, for example, G 
of 50, in this case. In addition, the value of R may be matched 
with the average value of G and B, that is, 40, in this case. 

[0033] 

Furthermore, if the color of the pupils do not match the 

color of the operator's preference, a desired color is selected 

from a color pallet (samples) for the pupil, and corrections are 

made based on this color data. As for the correction method, for 

example, the following procedures are followed. Color data, YIQ, 
for example, for the selected color are obtained by the following 
equation. 

[0034] 
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[0035] 

The maximum value of B + G for the red-eye pixels is 
obtained and indicated as Lmax. Brightness (Y) of the pallet is 
multiplied by the value obtained by dividing L (= B + G) of the 
target pixels by Lmax (refer to the following equation). 

Y' = Y X L/Lmax 

While keeping the values of I and Q, reverse conversion into 
R, G, and B is carried out to determine said digital value as a 
correction value, A more natural look can be achieved in this 
way. 

[0036] 

In other words, although the pallet is used select the 
color, because the corrections are made using the brightness data 
for the elements other than the red-eye pixels, a natural look 
can also be obtained. Although the chromaticity of the red color 
was obtained over a series of processing, combination of other 
general color data, such as brightness, hue, chroma, and so 
forth, may be used. The present embodiment offers the following 
effects . 

[0037] 

® Little involvement of the operator is needed. 

® Because edge data are utilized for the division of area, 
in comparison to the method in which color data alone are 
utilized for the division, influence by. the differences in terms 



of the photographic scene and the individuals is unlikely to 
occur, and correction can be made at high efficiency. 

(D Cases in which those other than Japanese people are 
involved where the color of the aforementioned pupils is not gray 
can also be dealt with by making the aforementioned corrections. 

[0038] 

Next, a second embodiment will be explained. In the 
aforementioned first embodiment, there are cases in which only a 
portion of the red-eye part to be corrected may be corrected due 
to the differences in red-eye parts among individuals, or the 
parts where red-eye is not present are also corrected. In the 
present embodiment, in order to prevent this problem, the 
operator adjusts the threshold for determining whether 
corrections are needed or not while looking at an image, so that 
a correction area can be adjusted. 

[0039] 

More specifically, if the operator determines that only a 
portion of a red-eye part is corrected when looking at the image, 
the value of the aforementioned threshold (TH) set for the 
detection of red-eye pixels is lowered repeatedly until the 
correction area is enlarged to an appropriate area. On the other 
hand, if the operator determines that the parts where red-eye is 
not present are also corrected, the value of the aforementioned 
threshold (TH) is increased repeatedly until the correction area 
is reduced to au appropriate area. 
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[0040] 

Designation of a red-eye area is partially dependent upon 
subjectivity, so that by doing it this way, correction which 
would satisfy the operator's subjectivity can be made. Also, 
variations in the color data for the red-eye areas and the 
surrounding areas due to the differences in terms of the 
photographic scene and the individuals can be absorbed. 

Brief description of the ^xgures 

Figure 1 is a block diagram showing the 
configuration/function of the present invention. 

Figure 2 is a diagram showing the system configuration of an 
embodiment . 

Figure 3 is a flow chart showing the main routine for the 
red-eye correction in the above embodiment. 

Figure 4 is a flow chart showing the subroutine for the 
division into small areas under the above routine. 

Figure 5 a flow chart showing the subroutine for the 
selection of pupil candidate areas under the above routine. 

Figure 6 is a diagram showing the target area designation 
method in the above embodiment . 

Symbols 
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Mempry unit 
Control unit 
Monitor 
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Figure 1 

Designation of area possibly containing poor color 
tones 

Division into small areas 
Designation of pupil candidate areas 
Extraction of pixels with poor color tones 
Correction of pixels with poor color tones 
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Digital color image data 
Memory unit 
Control unit 
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Figure 3 

Key: SI Designation of an area surrounding a pair of pupils, 
specification of a point between the eyes 

52 Calculation of the characteristic quantity at the 
specified point 

53 Extraction of a skin color area 

54 Calculation of the average value (Rskin) of the 
chromaticity of the red color in the skin color area 

55 Designation of an area circumscribing the skin color 
area 

56 Division into small areas 

57 Selection of pupil candidate areas 

58 Extraction of red-eye pixels 

59 Correction of red-eye pixels 
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Figure 4 



Key: a Routine for division into small areas 

511 Extraction of edge pixels 

512 Labeling of edge pixels 

513 Labeling of the area surrounded by the edge pixels 
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Figure 5 
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Routine for selection of pupil candidate areas 
The maximum value (Ri) of the chromaticity of the red 
color over the areas and the position (Xi, Yi) are 
obtained . 

Characteristic quantity of color data 
Ci2 = Ri + R2 

characteristic quantity of positional 
data = Xi + X2 + Y: + Y2 
P12 = Xa + X2 + Yi + Y2 

The one highest in £12 = C12 - P12 is selected as a 
candidate area. 
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Figure 6 
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Rectangular area 
Center point 



